Combinatorial Multi-armed Bandit Research Articles

A Quality-Aware and Obfuscation-Based Data Collection Scheme for Cyber-Physical Metaverse Systems

In pursuit of an immersive virtual experience within the Cyber-Physical Metaverse Systems (CPMS), the construction of Avatars often requires a significant amount of real-world data. Mobile Crowd Sensing (MCS) has emerged as an efficient method for collecting data for CPMS. While progress has been made in protecting the privacy of workers, little attention has been given to safeguarding task privacy, potentially exposing the intentions of applications and posing risks to the development of the Metaverse. Additionally, existing privacy protection schemes hinder the exchange of information among entities, inadvertently compromising the quality of the collected data. To this end, we propose a Quality-aware and Obfuscation-based Task Privacy-Preserving (QOTPP) scheme, which protects task privacy and enhances data quality without third-party involvement. The QOTPP scheme initially employs the insight of “showing the fake, and hiding the real” by employing differential privacy techniques to create fake tasks and conceal genuine ones. Additionally, we introduce a two-tier truth discovery mechanism using Deep Matrix Factorization (DMF) to efficiently identify high-quality workers. Furthermore, we propose a Combinatorial Multi-Armed Bandit (CMAB)-based worker incentive and selection mechanism to improve the quality of data collection. Theoretical analysis confirms that our QOTPP scheme satisfies essential properties such as truthfulness, individual rationality, and ϵ-differential privacy. Extensive simulation experiments validate the state-of-the-art performance achieved by QOTPP.

Combinatorial Stochastic-Greedy Bandit

We propose a novel combinatorial stochastic-greedy bandit (SGB) algorithm for combinatorial multi-armed bandit problems when no extra information other than the joint reward of the selected set of n arms at each time step t in [T] is observed. SGB adopts an optimized stochastic-explore-then-commit approach and is specifically designed for scenarios with a large set of base arms. Unlike existing methods that explore the entire set of unselected base arms during each selection step, our SGB algorithm samples only an optimized proportion of unselected arms and selects actions from this subset. We prove that our algorithm achieves a (1-1/e)-regret bound of O(n^(1/3) k^(2/3) T^(2/3) log(T)^(2/3)) for monotone stochastic submodular rewards, which outperforms the state-of-the-art in terms of the cardinality constraint k. Furthermore, we empirically evaluate the performance of our algorithm in the context of online constrained social influence maximization. Our results demonstrate that our proposed approach consistently outperforms the other algorithms, increasing the performance gap as k grows.

Immersive Multimedia Service Caching in Edge Cloud with Renewable Energy

Immersive service caching, based on the intelligent edge cloud, can meet delay-sensitive service requirements. Although numerous service caching solutions for edge clouds have been designed, they have not been well explored. Moreover, to the best of our knowledge, there is no work to consider the immersive service caching scheme under the supply of renewable energy. In this article, we investigate the service caching problem under the renewable energy supply to minimize service latency while making full use of renewable energy. Specifically, we formulate the service caching and renewable energy harvesting problem, which considers the dynamic renewable energy, unknown service requests, and limited capacity of the edge cloud. To solve this problem, we propose an effective algorithm, called OSCRE. Our algorithm first uses Lyapunov optimization to convert the time-average problem into time-independence optimization and thus realizes optimal renewable energy harvesting. Then, it realizes the service caching scheme using data-driven combinatorial multi-armed bandit learning. The simulation results show that the OSCRE scheme can save service latency while making sufficient use of renewable energy.

Multi-hop relay selection for underwater acoustic sensor networks: A dynamic combinatorial multi-armed bandit learning approach

An efficient multi-hop relay selection method is the key of multi-hop relay technology to improve wireless communication reliability. Accordingly, this paper devotes a multi-hop relay selection problem for unknown time-varying underwater acoustic sensor networks, and a dynamic combinatorial multi-armed bandit (DCMAB) learning structure is proposed to achieve the multi-hop relay strategy with minimum propagation delay without any prior channel information. Compared with the strategy learning space of the single relay selection problem for static networks, the multi-hop relay learning space shows high-dimensional and dynamic characteristics. To cope with the high-dimensional characteristic of multi-hop relay strategy spaces, DCMAB develops a combinatorial bandit learning manner. It enables the player to learn the high-dimensional multi-hop relay strategy space by exploring the low-dimensional link sub-strategy space, thereby reducing the learning complexity. To cope with the dynamic characteristic of multi-hop relay strategy spaces, DCMAB makes newly-formed links able to employ the historical learning information of experienced links to reason their prior knowledge. Meanwhile, by adopting a probabilistic compensation manner, DCMAB intensifies the exploration for newly-formed links. It successfully overcomes learning inefficiency caused by the lack of learning information on newly-formed links. Besides, an energy-aware-based filtering mechanism is proposed to filter out potential long-delay relay links. It enables the player to focus on exploring and reasoning high-quality links, thereby enhancing the quick search ability of superior multi-hop relay strategies. Finally, the superiority of the proposed algorithm is demonstrated by extensive simulation results.

Investigation of selection and application of Multi-Armed Bandit algorithms in recommendation system

The Multi-Armed Bandit (MAB) algorithm holds significant prominence as a recommendation system technique, effectively presenting user-centric content preferences based on the analysis of collected data. However, the application of the basic MAB algorithm in real-world recommendation systems is not without challenges, including issues related to data volume and data processing accuracy. Therefore, the optimization algorithm based on the MAB algorithm is more widely used in the recommendation system. This paper briefly introduces the multi-armed bandit algorithm, that is, the use of MAB in the recommendation system and the problems of the basic MAB algorithm. Aiming at the problems of the basic MAB algorithm, it introduces the MAB-based optimization algorithm used in the recommendation system. At the same time, this paper also analyzes and summarizes such algorithms. This paper introduces two different MAB-based optimization algorithms, namely The Details of Dynamic clustering based contextual combinatorial multi-armed bandit (DC3MAB) and Binary Upper Confidence Bound (BiUCB). In addition, this paper also introduces the application of algorithm in recommended system. Finally, this paper summarizes the introduced algorithms and proposes the future prospects for MAB optimization algorithms.

BTV-CMAB: A Bi-Directional Trust Verification-Based Combinatorial Multiarmed Bandit Scheme for Mobile Crowdsourcing

BTV-CMAB: A Bi-Directional Trust Verification-Based Combinatorial Multiarmed Bandit Scheme for Mobile Crowdsourcing

Thompson Sampling-Based Partially Observable Online Change Detection for Exponential Families

This paper proposes a holistic sequential change detection framework for partially observable high-dimensional data streams with exponential-family distributions. The framework first proposes a general composite decomposition for exponential-family distributed data by projecting its natural parameter onto normal bases and abnormal bases, which enables efficient inference for sparse changes. Then, the inference results are used for detection scheme construction, and different types of test statistics can be compacted in our framework. Last, by further designing the test statistic as the reward function in the combinatorial multi-armed bandit problem, a Thompson sampling-based sensor allocation strategy is constructed to select the most anomalous variables. Theoretical properties of the detection framework are discussed. Finally, examples of Gaussian, Poisson, and binomial distributed data streams are given in numerical studies and case studies to evaluate the performance of our proposed method. History: Bianca Maria Colosimo served as the senior editor. Funding: C. Zhang is partially supported by the NSFC [Grants 71932006 and 72271138], the BNSF [Grant 9222014], and the ASFC [Grant 2020Z063058001]. H. Yan is partially supported by NIH [R21 AI157618] and NSF [CMMI 2316654]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijds.2022.00011 . Data Ethics & Reproducibility Note: No data ethics considerations are foreseen related to this paper. The code capsule is available on Code Ocean at https://codeocean.com/capsule/8794940/tree/v1 and in the e-Companion to this article (available at https://doi.org/10.1287/ijds.2022.00011 ).

Adaptive Label Propagation for Group Anomaly Detection in Large-Scale Networks

This paper concentrates on group anomalies in general large-scale networks. Existing algorithms on group anomalies mainly focus on homogeneous or bipartite networks, and thus are difficult to apply to heterogeneous networks directly. Moreover, these algorithms follow the non-overlapping hypothesis of groups implicitly, which is improper in many scenarios. In this paper, we introduce a novel algorithm called Adaptive Label Propagation (ALP) to solve these problems. ALP is designed based on label propagation (LP) frameworks, for the reason that LP-based frameworks are simple in thought and easy to scale. ALP is able to find overlapping groups by label propagation with belonging coefficients, and can be applied to heterogeneous networks for its design of adaptive neighbor weighting. Assigning different weights to neighbors in label propagation is a challenging task. Inspired by the combinatorial multi-armed bandit mechanism, ALP views the neighbors of each node as arms to be selected, and iteratively updates their weights by evaluating their expected rewards in following iterations. Experiments are conducted on four real-world networks. The results show that LP-based methods are effective for detecting group anomalies, and the comparison results with several state-of-the-art label propagation based community detection methods show the effectiveness of the proposed method

FedAB: Truthful Federated Learning With Auction-Based Combinatorial Multi-Armed Bandit

Federated Learning (FL) emerges as a new distributed machine learning (ML) paradigm that enables thousands of mobile devices to collaboratively train ML models using local data without compromising user privacy. However, the FL learning quality highly relies on the data contribution from the distributed mobile devices. Therefore, a well-designed incentive mechanism with effectiveness, fairness, and reciprocity is in urgent need to guarantee the stable participation of users. In this paper, we propose FedAB, an incentive and client selection strategy based on a novel multi-attribute reverse auction mechanism and a combinatorial multi-armed bandit (CMAB) algorithm. First, we develop a local contribution evaluation method based on importance sampling in the FL context. We then design a novel payment mechanism that is able to preserve individual rationality and incentive compatibility (truthfulness). At last, we design a UCB-based winner selection algorithm that is proven to achieve the server’s utility maximization with fairness and reciprocity. We have conducted extensive experiments on real datasets. The results demonstrate the superiority of FedAB, with a 10-50% improvement in total reward, final accuracy, and convergence speed compared to state-of-the-art solutions.

Spectrum Allocation and User Scheduling Based on Combinatorial Multi-Armed Bandit for 5G Massive MIMO.

As a key 5G technology, massive multiple-input multiple-output (MIMO) can effectively improve system capacity and reduce latency. This paper proposes a user scheduling and spectrum allocation method based on combinatorial multi-armed bandit (CMAB) for a massive MIMO system. Compared with traditional methods, the proposed CMAB-based method can avoid channel estimation for all users, significantly reduce pilot overhead, and improve spectral efficiency. Specifically, the proposed method is a two-stage method; in the first stage, we transform the user scheduling problem into a CMAB problem, with each user being referred to as a base arm and the energy of the channel being considered a reward. A linear upper confidence bound (UCB) arm selection algorithm is proposed. It is proved that the proposed user scheduling algorithm experiences logarithmic regret over time. In the second stage, by grouping the statistical channel state information (CSI), such that the statistical CSI of the users in the angular domain in different groups is approximately orthogonal, we are able to select one user in each group and allocate a subcarrier to the selected users, so that the channels of users on each subcarrier are approximately orthogonal, which can reduce the inter-user interference and improve the spectral efficiency. The simulation results validate that the proposed method has a high spectral efficiency.

Truth based three-tier Combinatorial Multi-Armed Bandit ecosystems for mobile crowdsensing

Many Multi-Armed Bandit (MAB) based workers selection schemes have been proposed to select high-quality workers to enhance the quality of tasks. However, in Mobile Crowd Sensing (MCS), a complex mutual effect exists among task requestors, the MCS platforms, and workers. Only considering the interaction of two sides doesn’t make MCS a balanced ecosystem. Therefore, it is urgent to establish a tripartite mutual incentive mechanism to make the MCS system a balanced ecosystem. In this paper, a truth based Three-tier Combinatorial Multi-Armed Bandit (TCMAB) incentive mechanism is proposed for selecting each other to maximize their revenues in MCS. In TCMAB, there exists a three-tier and two-way MAB-based incentive scheme. For the mutual interaction between the platform and the worker, a truth-based CMAB scheme is established for the platform to select high-quality workers, and also a CMAB scheme is proposed for workers to select a “good” platform to report data in order to maximize their revenue. Besides, for the mutual interaction between the task requestor and the platform, the platforms adopt the CMAB-based scheme to select a high-payment task requestor that gives high payment. And a platform selection scheme base on CMAB is also established for task requestors to select the platforms which have lower fees and higher quality. What’s more, we don’t adopt the assumption that platforms get the data quality as soon as they get data, but propose a data quality acquisition scheme based on the truth data discovery and cooperation frequency, which is the base to instruct the three-tier interaction, thus establish a kind of truth-based MCS interaction ecosystems. Simulation results show that the proposed TCMAB provides an effective solution for the problem of information elicitation without verification (IEWV) in MCS, and can improve the utilities, data quality, and applications quality for MCS, which is not achieved in the previous studies significantly.

LEO satellite assisted UAV distribution using combinatorial bandit with fairness and budget constraints.

In this paper, an integration between a low earth orbit satellite (LEO-Sat) and unmanned aerial vehicle (UAV) is proposed to assist users in post-disaster areas. In this scenario, multiple UAVs will be distributed to fully cover the victims and provide rescue services, while LEO-Sat provides backhaul links for UAVs to the ground base station (GBS). In this regard, we consider the problem of efficient UAVs distribution to maximize the total sum rate of the victims while assuring fairness in their coverage within the limited resources of UAVs batteries and LEO-Sat bandwidth. In this paper, UAV distribution problem is considered as a combinatorial multi-armed bandit (MAB) with arms' fairness and limited UAVs battery budget (CMAB-FB) constraints. Additionally, the utilization of LEO-Sat bandwidth resources is optimized based on the average traffic demands of the LEO-UAV links by means of gradient decent algorithm. The results of numerical analysis indicate that the proposed approach outperforms other naïve ben chmarks.

Federated Online Learning Aided Multi-Objective Proactive Caching in Heterogeneous Edge Networks

To address the drastic increase in multimedia traffic volume, mobile edge caching (MEC) has been exploited to reduce redundant data transmissions by equipping computation and storage capacity at the edge network. Previous works on learning-based caching problems often only concern pre-storing popular contents to satisfy users’ demands. In this work, we investigate the cache strategy design problem with two possibly conflicting objectives, namely, cache hit and cache profit, in heterogeneous multi-MEC server networks when content profiles are unknown. We then formulate this multi-objective caching problem as a Multi-agent Multi-objective Combinatorial Multi-Armed bandit (MMC-MAB) problem and propose a two-step caching framework that estimates content properties first and then optimizes cache placement. Specifically, to accommodate the system heterogeneity in estimation, we utilize an adaptive federated learning-based estimation to approach the unknown content popularity and profit profiles, which wisely use both local and external observations by adjusting the mixing factors. To address the multiple objective optimizations, we propose two effective methods, based on individual dominance and combinatorial dominance, to achieve adequate Pareto-optimal cache placement. Both theoretical results and comprehensive experiments clearly validate the effectiveness and efficiency of our proposed approaches.

Incentive-driven long-term optimization for hierarchical federated learning

Hierarchical federated learning (HFL), an emerging paradigm of the client-edge-cloud architecture, can effectively leverage nearby edge servers to conduct model aggregation, significantly reducing transmission overhead. HFL faces both technical and economic challenges: first, in the online setting, computation resources and network bandwidth can only reveal themselves when clients participate in HFL model training. Second, the model training process consumes substantial resources at the clients, such as energy, computation, and bandwidth. It is unrealistic to assume that all clients contribute their resources voluntarily. Thorough investigation is lacking for these challenges in existing HFL research. This work develops a novel online algorithm AUCS, based on the auction and combinatorial multi-armed bandit, to minimize the overall latency of HFL training. AUCS utilizes the ratio of upper confidence bound-based reward to the bid as a criterion for winner determination. Then, AUCS computes the key payment for each winner to guarantee truthfulness of the incentive mechanism. Theoretically, AUCS can achieve sub-linear regret, truthfulness, individual rationality and computational efficiency, and guarantees model convergence. Simulations on real-world datasets and training tasks demonstrate the advantages of AUCS in terms of training latency, model accuracy, and system efficiency.

Modelling underreported spatio-temporal crime events.

Crime observations are one of the principal inputs used by governments for designing citizens' security strategies. However, crime measurements are obscured by underreporting biases, resulting in the so-called "dark figure of crime". This work studies the possibility of recovering "true" crime and underreported incident rates over time using sequentially available daily data. For this, a novel underreporting model of spatiotemporal events based on the combinatorial multi-armed bandit framework was proposed. Through extensive simulations, the proposed methodology was validated for identifying the fundamental parameters of the proposed model: the "true" rates of incidence and underreporting of events. Once the proposed model was validated, crime data from a large city, Bogotá (Colombia), was used to estimate the "true" crime and underreporting rates. Our results suggest that this methodology could be used to rapidly estimate the underreporting rates of spatiotemporal events, which is a critical problem in public policy design.

Truthful User Recruitment for Cooperative Crowdsensing Task: A Combinatorial Multi-Armed Bandit Approach

Mobile Crowdsensing (MCS) is a promising paradigm that recruits users to cooperatively perform a sensing task. When recruiting users, existing works mainly focus on users objective abilities, e.g., the probability or frequency of covering the task locations. However, we argue that the task completion effect depends not only on the user's objective ability, but also on their subjective collaboration likelihood with each other. Even though we can find a well-behaved group of users in the single-round scenario, while in the multi-round scenario without enough prior knowledge, we still face the problem of exploiting users or exploring users. Additionally, we consider that users have different costs and may report false costs to gain more benefits. To address these problems, we first convert the single-round user recruitment problem into the min-cut problem and propose a graph theory based algorithm to find the approximate solution. Then, in the multi-round scenario, we propose a multi-round User Recruitment strategy under the budget constraint based on the combinatorial Multi-armed Bandit model (URMB), which achieves a tight regret bound. Next, we propose a graph-based payment strategy to ensure the truthfulness and individual rationality. Finally, experiments based on real-world datasets show that URMB always outperforms the state-of-the-art strategies.

Dynamic User Recruitment in Edge-Aided Mobile Crowdsensing

Most of the existing user recruitment methods are designed for the two-tier structure. However, the computational consumption of the cloud platform rises rapidly as data volumes increase. In recent years, edge computing has been introduced into mobile crowdsensing. In this paper, we study dynamic user recruitment with edge-aided mobile crowdsensing. Specifically, we consider the scenarios for different sensing tasks. One is the long-duration task, and the other is the short-duration task. For the former, it is time-insensitive, while the latter is time-sensitive. In the scenario with the long-duration task, we consider a simple offline scenario and propose an Edge-Node User Recruitment (ENUR) algorithm to recruit users statically. Then, we consider the online scenario and propose a Budget Re-Distribution algorithm for Edge Nodes User Recruitment (BRD-ENUR). It dynamically redistributes the budget according to the task completion ratio of different edge nodes. In the scenario with the short-duration task, the whole sensing process consists of multiple rounds. In each round, edge nodes need to recruit users between selected users (exploitation) and unselected users (exploration) under the budget. We model such a user recruitment process as a novel combinatorial multi-armed bandit problem. Then, we propose an Edge Nodes User Recruitment algorithm based on combinatorial Multi-armed Bandit (EN-URMB). Extensive experiments show the validity and reliability of our user recruitment algorithms based on the real data set.

Reliability-Aware Online Scheduling for DNN Inference Tasks in Mobile-Edge Computing

Mobile Edge Computing (MEC) is widely envisioned as a promising technique for provisioning artificial intelligence (AI) capability for resource-limited Internet of Things (IoT) devices by leveraging edge servers (ESs) for executing Deep Neural Network (DNN) inference tasks in proximity. However, scheduling DNN inference tasks at the network edge under unknown system dynamics (e.g., uncertain availability of ESs) may suffer from failures, making it difficult to guarantee reliable services for the IoT device. To overcome this challenge, we propose a reliability-aware online scheduling scheme for DNN inference tasks in MEC by leveraging both online feedback and offline data to learn the uncertain availability of ESs to maximize both the inference accuracy and service reliability of DNN inference tasks (i.e., the number of DNN inference tasks processed during the system span). We first formulate the reliability-aware DNN inference tasks scheduling problem as a novel constrained combinatorial multi-armed bandit (CMAB) problem. Then by integrating the Lyapunov optimization technique, bandit learning, approximated submodular maximization, and historical data organically, we design a Reliability-Aware Task scheduling scheme with Bandit Learning (RTBL) algorithm to solve this problem. Unfortunately, even with an accurate prediction of the system uncertainties, the task scheduling problem is still NP-hard. To deal with it, we therefore design an advanced approximation algorithm based on the submodularity of the scheduling problem which obtains a near-optimal solution and provides a satisfactory performance guarantee. Finally, we conduct rigorous theoretical analysis and race-driven simulations to show RTBL’s brilliant performance.

A combinatorial multi-armed bandit approach to correlation clustering

Given a graph whose edges are assigned positive-type and negative-type weights, the problem of correlation clustering aims at grouping the graph vertices so as to minimize (resp. maximize) the sum of negative-type (resp. positive-type) intra-cluster weights plus the sum of positive-type (resp. negative-type) inter-cluster weights. In correlation clustering, it is typically assumed that the weights are readily available. This is a rather strong hypothesis, which is unrealistic in several scenarios. To overcome this limitation, in this work we focus on the setting where edge weights of a correlation-clustering instance are unknown, and they have to be estimated in multiple rounds, while performing the clustering. The clustering solutions produced in the various rounds provide a feedback to properly adjust the weight estimates, and the goal is to maximize the cumulative quality of the clusterings. We tackle this problem by resorting to the reinforcement-learning paradigm, and, specifically, we design for the first time a Combinatorial Multi-Armed Bandit (CMAB) framework for correlation clustering. We provide a variety of contributions, namely (1) formulations of the minimization and maximization variants of correlation clustering in a CMAB setting; (2) adaptation of well-established CMAB algorithms to the correlation-clustering context; (3) regret analyses to theoretically bound the accuracy of these algorithms; (4) design of further (heuristic) algorithms to have the probability constraint satisfied at every round (key condition to soundly adopt efficient yet effective algorithms for correlation clustering as CMAB oracles); (5) extensive experimental comparison among a variety of both CMAB and non-CMAB approaches for correlation clustering.

Two Stage Beamforming in Massive MIMO: A Combinatorial Multi-Armed Bandit Based Approach

In frequency division duplex (FDD) massive multi-input multi-output (MIMO), the two-stage beamforming (TSB) using channel covariance matrices (CCM) can significantly reduce the downlink training length (DTL) and channel feedback. However, the overhead to estimate the CCM is large. In this paper, a combinatorial multi-armed bandit (CMAB) based TSB scheme is proposed without requirement of CMM. Particularly, the problem of the pre-beamforming matrix design is transformed into a CMAB problem. We consider the pre-beamforming matrix design in each slot as the arm selection in the CMAB, and convert the problem of the arm selection into a 0-1 integer linear programming problem, which can be solved by the branch-and-bound method. During the training process, the maximum likelihood method is used to detect the power of angle spectrum, and the angle range of each user is determined adaptively. We prove that the regret grows logarithmically with time, such that the proposed scheme converges towards the optimal action. Finally, simulation results demonstrate that the proposed scheme can significantly improve the spectral efficiency.